Experimental Design and Testing of a Multimodal Cognitive Overload Classifier

We report the results of an experiment designed to construct and test a robust multimodal system for automatic classification of cognitive overload. With the assistance of The Scripps Research Institute, twenty-two experienced gamers performed a first-person shooter combat simulation while multiple biosignals and performance were recorded. Signals included EEG, EOG, EMG, ECG, accuracy, and reaction time measures. A kernel partial least squares or KPLS classifier was trained to distinguish subtle differences in EEG spectra within subjects as they pertained to passive viewing, low-difficulty, and high-difficulty simulations. The KPLS classifier was supported and enhanced by algorithms for preprocessing and normalization of EEG and other biosignals. Results indicated that for some subjects, robust classifiers discriminated passive viewing from active performance with accuracies in the range of 99% to 100% and that such models were stable over two test days, using 35-70 min. of training data from Day 1 and testing on data from Day 2. In addition, for some subjects, classifiers discriminated high-difficulty simulations from low-difficulty and passive simulations with stable accuracies of 80% or better across two test days, using 35-70 min. of training data from Day 1 and testing on data from Day 2. We also tested the effect of alcohol intoxication on simulation performance and classifier accuracy. Performance was slightly altered by drinking alcohol to a blood alcohol level of 0.06%, producing more aggressive behavior than with a placebo across subjects. The KPLS classifiers showed remarkable resilience to alcohol effects, considerably less than the effects of day of testing. Not all subjects had such impressive results. To address the lack of generality across subjects, we are currently performing a modified study design that includes provisions for three calibration tasks. These calibration tasks are intended to serve as brief, simple tasks that a user could easily perform at any time as needed to recalibrate algorithms that relate physiology to cognitive state. The tasks include a) eyes-open and eyes-closed for general EEG calibration, b) a mental arithmetic task (divide by twos or sevens) for cognitive load classification, and c) passive viewing of the combat simulation task, for calibration of engagement/disengagement of mental resources. We aim to use the data from the calibration tasks to adapt classification algorithms for variations over time and for inclusion of multiple sensor and data modalities, such as electrocardiographic, electrooculographic, and electromyographic sensor data.